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Generative  Computer  Assisted  Instruction 
and  Artificial  Intelligence 


INTRODUCTION 

As  has  often  been  noted  (.e.g.,  Carbonell,  1970a;  Uttal,  Pasich,  Rogers,  & 
Hieronymous,  1969),  traditional  frame-oriented  computer-assisted  instruction 
does  not  make  full  use  of  the  capabilities  of  Lhe  computer.  In  frame-oriented 
CAL  students  are  presented  prestored  lext  and  text-related  questions.  Although 
complex  branching  strategies  may  be  employed,  frame-oriented  CAI  is  basically  a 
mechanized  version  of  a programmed  text.  As  such  it  makes  maximal  use  of  the 
computer's  capacity  for  storing  large  amounts  of  data,  hut  minimal  use  of 
the  computer's  logical  capabilities. 

Certain  modes  of  computer-assisted  instruction,  such  as  tutorial  and 
drill-and-practice , have  been  successfully  implemented  in  frame-oriented 

environments  ( Rockavt  & Scott  Morton,  1975).  Rut  frame-oriented  systems 

cannot  effectively  handle  other  forms  of  CAI  applications,  such  as  those 
supporting  problem  solving  activities  or  explorations  of  social  or  physical 
system  simulations.  These  forms  require  a program  which  can  do  more  than 
literal  manipulation  of  text  in  sequences  predetermined  by  the  program's 
author;  they  require  a system  which  can  adapt  to  specific,  unanticipated 
interactions  with  the  student. 

The  class  of  CAI  programs  termed  "generative"  encompasses  systems  which 
can  interact  responsively  in  less  predictable  modes  of  CAI,  like  problem 
solving  or  computer  simulations.  Through  programmed  procedures,  generative 
systems  bring  together  elements  of  data  bases  to  construct,  for  example, 
answers  to  questions  posed  by  the  student,  segments  of  instructional  text, 
questions  directed  to  the  student,  or  answers  to  computer  constructed  questions. 

A trend  in  the  development  of  generative  CAI  has  been  to  develop  systems 
which  act  much  like  intelligent  human  tutors.  Since  a goal  of  artifical 
intelligence  research  is  "to  construct  programs  which  exhibit  what  we  call 
Intelligent  behavior"  (Felgenbaum  & Feldman,  1963),  A1  is  the  breeding 
ground  for  techniques  applied  in  generative  systems  which  aid  in  the 
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achievement  of  this  tutor-like  environment.  Implementation  of  techniques 
from  AI  has  produced  a new  level  of  sophistication  in  CAI  applications, 
providing  environments  only  partially  described  by  the  term  "generative." 

A more  descriptive  designation  for  these  systems,  stemming  from  their  emphasis 
on  tutor  modeling,  is  "responsive"  computer-assisted  instruction. 

PERSPECTIVE 

In  1969,  Uttal  ot  al.  described  generative  programs  as  those  which  use 
"algorithms  to  generate  problems,  answers,  diagnostics,  and  remedial  materials." 
At  the  time,  they  felt  that  not  all  subject  domains  were  amenable  to  generative 
development,  just  those  conducive  to  algorithmic  manipulation,  such  as 
'mathematical  course  materials,  other  mathematically  oriented  subjects  like 
the  physical  sciences,  and  some  special  subject  matters  such  as  chemistry  or 
logic  in  which  there  is  a formal  system  describing  the  relations  between 
and  among  parts."  In  particular,  they  excluded  "verbally  oriented  subject 
matter  as  possible  items  for  a generative  curriculum." 

A year  later,  two  systems  emerged  which  indicated  that  verbally  oriented 
subject  matter  could  be  taught  in  a generative  environment  (Carbonell,  1970a; 
Wexler,  1970).  Both  were  tutors  ol  geography.  Though  applicable  to  other 

subject  areas,  both  were  oriented  toward  the  tutoring  of  factual  knowledge. 

The  systems  used  structured  information  networks  to  store  key  concepts 
in  their  subject  domain.  But  the  locus  of  the  meaning  of  those  concepts 
differed  between  the  programs.  Meaning  in  Wexler' s system  resided,  for  the 
most  part,  in  a prestored  set  of  incomplete  sentences.  Concepts  were 
retrieved  from  the  network  to  appropriately  complete  the  statements,  which 
were  then  transmitted  to  the  student. 

The  emphasis  of  meaning  In  Carbonell 's  system  resided  in  the  network 
itself.  Following  the  work  of  Quillian  (1968)  in  natural  language  comprehension, 
Carbonell  constructed  a detailed  network  of  interrelated  concepts  stored  in 
attribute-value  (e.g.,  Capltal-Santiago)  format.  Facts  about  a geographical 
concept  (e.g.,  Argentina)  could  be  Inferred  from  the  network  by  procedures 
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which  interpreted  a list  of  properties  associated  with  the  concept,  each 
property  consisting  of  an  attribute  and  its  value  or  set  of  values.  The 
values  in  turn  had  their  own  associated  property  lists,  allowing  for  multiple 
embedding  of  information.  No  pieces  of  text  or  questions  were  part  of  the 
network.  Procedures,  almost  completely  independent  of  the  subject  matter 
to  which  they  applied,  were  capable  of  searching  the  network  to  construct 
the  output  the  system  presented  to  the  student. 

Although  Wexler  provided  a generative  tutor  in  a subject  considerably 
less  formal  than  those  Uttal  and  his  associates  thought  possible,  his  system 
was  not  a theoretical  departure  from  the  notion  of  algorithmic  generation 
held  by  these  investigators.  On  the  other  hand,  Carbonell's  signified  a new 
direction  in  the  evolution  of  generative  programs.  By  endowing  SCHOLAR,  as 
the  system  was  called,  with  a representation  of  its  subject  matter  in  the  form 
of  a network  and  procedures  which  understood  it,  Carbonell  took  the  step 
toward  providing  CAI  programs  with  independent  problem  solving  and  question 
answering  expertise.  Carbonell  borrowed  techniques  from  artificial  intelligence, 
specifically  from  the  area  of  natural  language  comprehension,  to  give  SCHOLAR 
some  'understanding'  of  geography. 

Since  SCHOLAR'S  development,  applications  of  AI  in  CAI  have  continued, 

with  exemplary  programs  including  a logic  teaching  system  (Goldberg,  1973), 

a system  for  teaching  integration  (Kimball,  1973),  a tutor  of  a text-editing 

system  (Grignetti,  Gould,  Hausmanti,  Bell,  Harris,  N Passafiume,  1974),  a 

laboratory  in  electronic  troubleshooting  (Brown,  Burton,  & Bell,  1974a),  a 

tutor  of  a machine  language  (Koffman  .v  Blount,  1974,  1975),  and  a tutor  of 

BASIC  (Barr,  Beard,  N Atkinson,  1975a>.  I’lio  following  statement  by  Brown, 

Bell,  and  ,’dvhe  I (19711  roll,  els  the  mulortving  philosophy  o(  this  approach: 

It  seems  almost  axiomatic  that  the  more  a CAI  system 
'understands'  about  its  subject  area,  the  more  effective 
.i  teaching  system  it  will  he.  Hven  more  important  is  the 
tact  that  systems  which  possess  such  understanding  can 
provide  a qualitatively  new  lev.  1 of  interaction  with 
the  student. 


STATE-OF-THE-ART  AND  ARPA  RESEARCH 
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ARPA  has  supported  research  in  the  development  of  responsive  CAI 
environments.  The  research  has  produced  both  practical  and  theoretical 
accomplishments . On  the  practical  side  can  be  listed  the  development  of 
tutors  which  have  the  potential  to  effectively  and  efficiently  train 
military  personnel.  On  the  theoretical  side  can  be  listed  advances  in  the 
modeling  of  subject  matter,  the  modeling  of  the  learner's  state  of  knowledge, 
and  the  modeling  of  teaching  strategies.  The  following  ARPA  supported 
projects  will  be  discussed  as  they  relate  to  these  accomplishments : 


and 


(1)  The  work  of  brown  and  associates  at  Bolt,  Beranek,  and  Newman, 
Inc.,  on  instructional  environments  for  problem  solving  and 
gaming . 

(2)  The  work  of  Collins  and  associates,  also  at  BBN,  on  research 
with  SCHOLAR-like  systems. 

(3)  The  work  of  Atkinson  and  associates  at  Stanford  University  on 
an  instructional  system  for  teaching  the  BASIC  programming 
language . 

(A)  The  work  of  Norman  and  associates  at  the  University  of 
California  at  Irvine  on  human  information  processing. 


Modeling  the  Subject  Matter 

The  capability  of  an  automated  tutor  to  respond  to  the  needs  of  its 
student  is  greatly  enhanced  if  the  tutor  can  interpret  and  pertinently  reply 
to  unanticipated  student  input.  If  the  tutor  is  capable  of  this,  the  student 
can,  for  example,  ask  for  clarification  of  some  previously  discussed  point  or 
fill  a critical  gap  in  his  knowledge  bv  querying  the  tutor.  Furthermore, 
the  range  of  student  answers  to  questions  the  computer  poses  need  not  match 
a finite  set  of  prestored  statements;  the  computer  can  provide  pertinent 
responses  to  many  unforeseen  answers. 

Programs  which  can  flexibly  and  intelligently  handle  unanticipated  input 
generally  require  a representation  of  the  content  of  their  subject  area  to 
which  they  can  turn  to  interpret  the  meaning  of  the  input  and  to  construct 
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a relevant  response  (Nilsson,  197A) . This  requirement  has  inspired  work 
on  modeling  representations  of  knowledge  for  use  in  automated  tutors. 

Subject  matter  modeling  has  typically  been  directed  at  modeling  specific 
subject  areas,  but  more  general  representations  have  been  attempted 
(e.g.,  Kingsley  & Stelzer,  1974). 

SOPHIE,  a tutor  developed  bv  Brown  and  coworkers,  can  meaningfully 
react  to  unanticipated  ideas  concerning  what  is  wrong  with  a faulted  piece 
of  electronic  equipment  (Brown  & Burton,  L975).  Basic  to  SOPHiE's 
ability  to  do  this  are  several  representations  of  the  subject  area,  each 
of  which  supports  different  aspects  of  SOPHiE's  interactions  with  students. 
Included  in  the  representations  called  upon  by  SOPHIE  are  (a)  a semantically- 
centered  parser  of  student  input,  (b)  a network  which  encodes  time-invariant 
factual  knowledge  about  the  instrument's  circuit,  (c)  a model  which  can 
simulate  the  behavior  of  the  instrument,  and  (d)  an  assortment  of  heuristic 
procedures,  each  of  which  specializes  in  inferring  specific  types  of  information 
from  the  results  of  the  circuit  simulation.  Before  discussing  these  represen- 
tations in  more  detail,  a description  of  the  purpose  and  appearance  of 
SOPHiE's  tutorial  interaction  will  be  given. 

SOPHiE's  goal  is  to  teach  the  qualitative  understanding  of  electronic 
circuits  necessary  for  troubleshooting.  To  this  end  SOPHIE  interacts  with 
the  student  while  the  student  debugs  a malfunctioning  piece  of  electronic 
equipment.  Its  success  at  supporting  this  interaction  has  led  to  SOPHiE's 
evaluation  as  the  most  impressive  artifical  intelligence-based  system  yet 
developed  (Bundersoti  & Faust,  1976). 

The  tutorial  begins  with  the  presentation  of  a schematic  of  the 
component's  circuit  and  a partial  specification  of  the  equipment's  symptoms 
for  some  particular  but  unknown  fault.  The  student  is  then  permitted  to 
examine  the  behavior  of  the  instrument  by  requesting  certain  measurements  of 
its  output  under  various  control  settings  or  requesting  tests  at  internal 
points  of  the  circuit.  The  student  can  ask  SOPHIE  specific  questions  about 
the  implications  of  a measurement.  He  can  ask  SOPHIE  advise  about  what  to 
consider  next  given  what  has  thus  far  been  discovered  about  the  circuit. 

SOPHIE  can  evaluate  a student's  hypothesis  about  what  is  wrong  with  the 
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circuit,  taking  into  consideration  all  the  information  the  student  should 
h3ve  been  able  to  derive  from  his  previous  measurements.  SOPHIE  can  also 
judge  the  merits  of  any  particular  measurement  request  in  view  of  the  prior 
sequence  of  measurements  the  student  has  made. 

A portion  of  a SOPHIE-student  interaction  taken  from  Brown,  Burton, 
Bell  and  Bobrow  (1974)  is  presented  in  Figure  1.  Although  the  dialogue  is 
difficult  to  fully  appreciate  without  some  prior  knowledge  of  electronic 
circuits,  it  should  still  provide  some  idea  of  how  flexible  and  seemingly 
intelligent  SOPHIE  is. 

Figure  1 


Segment  of  a SOPHIE-student  interaction  taken  from  a longer 
protocol  presented  in  Brown,  Burton,  B< ■: 1 , and  Bobrow  (1974) 
Statements  entered  by  the  student  are  underlined. 


WHAT  IS  THE  BASE  EMITTER  VOLTAGE  OF  THE  VOL  1 AGE  l 1MITINC  TRANSISTOR? 


THE  BASE  EMITTER  VOLTAGE  OF  Q5  IS  -.037  MILLIVOLTS. 
WHAT  ABOUT  FOR  THE  CURRENT  LIMITING  TRANSISTOR? 


THE  BASE  EMITTER  VOLTAGE  OF  Q6  IS  .349  VOLTS. 

WHAT  SHOULD  IT  BE? 

IN  A WORKING  CIRCUIT  THE  BASE  EMITTER  VOLTAGE  OF  Q6  IS  .593  VOLTS. 
REPLACE  06. 


I AM  GOING  TO  ASK  YOU  SOME  QUESTIONS  ABOUT  HOW  Q6  IS  FAULTED. 
ARE  ANY  JUNCTIONS  SHORTED? 

I DON'T  KNOW. 

ARE  ANY  TERMINAL, S OPEN? 


WHICH  TERMINALS  ARE  OPEN? 

THE  BASE. 

Q6  IS  NOT  FAULTED  IN  THAT  MANNER.  Qh  NOT  REPLACED. 

IS  IT  POSSIBLE  THAT  Q6  HAS^ AN  OPEN  BAS F ? 

IS  THE  COMPLETE  FAULT  MODE  EB/EC/BC  = OP/OK/OP  WHAT  YOU  INTENDED? 


WHEN  THE  SETTINGS  WERE:  CC=1.0,  CR=HIGH , LOAD=20,  VC=1.0,  VR=HIGH 

THAT  FAULT  CONFLICTS  WITH  THE  FOLLOWING  MEASUREMENT.  YOU  OBSERVED 
THAT  THE  OUTPUT  VOLTAGE  WAS  11.7  VOLTS.  IF  Q6  HAD  FAULT  MODE 
EB/EC/BC  = OP/OK/OP  IT  WOULD  BE  23.9  VOLTS.  IN  A WORKING  CIRCUIT 
IT  IS  19.9  VOLTS.  ALSO,  THAT  FAULT  CONFLICTS  WITH  THE  FOLLOWING 
MEASUREMENT.  YOU  OBSERVED  THAT  THE  BASE  EMITTER  VOLTAGE  OF  Q6  WAS  .349 
VOLTS.  IF  Q6  HAD  FAULT  MODE  EB/F.C/BC  = OP/OK/OP  IT  WOULD  BE  .712  VOLTS. 

IN  A WORKING  CIRCUIT  IT  IS  .593  VOLTS  THAT  FAULT  DOES  NOT  EXPLAIN  ANY  OF 
THE  SYMPTOMS  YOU  OBSERVED  BUT  IT  IS  CONSISTENT  WITH  THE  VALUES  FOR  THE  BASE 
EMITTER  VOLTAGE  OF  Q5  AND  THE  VOLTAGE  AT  N1 . 

In  SOPHIE,  student  input  is  converted  to  executable  LISP  code  by  a 
semantically  based  parser  (Brown,  Burton,  Bell  & Bobrow,  1974;  Burton, 

1974);  evaluation  of  the  code  provides  the  data  which  comprises  SOPHIE' s 
response  to  the  input.  A semantically  based  parser  is  to  be  distinguished 
from  a syntactically  based  parser  (like,  for  example,  an  augmented  transition 
network  parser  (Woods,  1970)).  In  syntactic  parsers,  substrings  of  input  are 
associated  with  syntactic  categories  such  as  "noun",  "verb",  "noun  phrase" 
or  "adjective".  In  semantic  parsers  substrings  are  related  to  conceptual 
categories  such  as  (in  the  case  of  SOPHIE)  "a  request  for  information", 

"a  command  to  change  a control  setting",  "a  measurement"  or  "a  location". 

In  general  a semantic  parser  could  lead  to  a phenomenal  proliferation 
of  categories  to  be  captured  by  the  grammar.  But  in  the  area  of  electronic 
troubleshooting  the  number  of  concepts  is  limited  enough  and  the  constraints  that 
interrelate  them  are  well  defined  enough  to  make  the  use  of  such  a parser 
feasible.  The  constraints  built  into  the  grammar  between  the  various 
conceptual  entities  which  SOPHIE  must  comprehend  represents  one  dimension  of 
SOPHIE's  multidimensional  "knowledge"  of  its  subject  matter. 

SOPHIE  derives  its  factual  knowledge  about  the  circuit  from  a network 
representation  of  this  information.  The  network  stores  only  time-invariant 
information  such  as  the  specifications  of  circuit  components  or  definitions 
of  circuit  terminology.  The  structure  of  the  network  is  in  the  tradition  of 
Quillian  (1969).  SOPHIE  calls  upon  the  network  when  it  answers  factual 
questions  about  the  circuit  posed  by  the  student.  The  network  is  also  called 
upon  by  SOPHIE's  heuristic  procedures  and  natural  language  processor  when 
factual  information  is  required  for  their  operation. 
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But  SOPHIE  must  also  deal  with  time-variant  information  about  the 
circuit  since  the  circuit's  state  is  constantly  undergoing  modifications 
during  the  course  of  troubleshooting.  It  would  be  impossible  to  provide 
SOPHIE  with  an  explicit  representation  of  all  potential  circuit  states  since 
there  are  an  indefinite  number  of  them.  Instead  SOPHIE  has  access  to  an 
implicit  representation  of  all  potential  states:  a model  which  simulates 

the  real  circuit's  behavior.  When  SOPHIE  requires  information  about  the 
circuit  in  some  specified  condition,  SOPHIE  activates  the  simulation  to 
produce  a circuit  representation  in  the  required  state.  SOPHIE  then  has  at 
its  disposal  a description  of  the  modified  circuit  from  which  it  may 
gather  the  information  it  needs.  Grignetti  et  al. , (1974)  draw  the  following 
analogy  when  observing  how  natural  it  is  to  use  a simulation  model  rn  a 
CAI  application:  "A  person's  data  base  is  not  only  memory,  and  his  'retrieval 

routines'  are  not  solely  introspective:  he  uses  the  world  as  a data  base 

and  his  senses  to  retrieve  information  from  it.  I don't  need  to  have  in 
my  head  a representation  of  what  is  behind  my  chair;  If  I need  to  know,  I 
can  just  turn  around,  look,  and  see!" 

According  to  Brown,  Burton,  and  Bell  (1974),  "the  main  seat  of  intelligence 
in  SOPHIE  resides  in  its  ability  to  draw  conclusions  and  make  inferences 
from  setting  up,  running,  and  examining  the  simulation  model  of  its  problem 
domain."  The  availability  of  the  circuit  simulation  and  heuristic  procedures 
which  can  interpret  it  is  responsible  for  the  success  with  which  SOPHIE 
handles  the  following  communications:  answers  to  questions  about  the  state 

of  the  circuit  after  some  modification  of  the  circuit,  evaluations  of  student 
hypotheses  concerning  the  equipment's  fault,  and  suggestions  of  hypotheses 
that  the  student  might  consider. 

More  generally,  for  subject  domains  amenable  to  such  modeling,  the  use 
of  a simulation  can  equip  a tutor  with  the  means  to  make  powerful  inferences. 

For  domains  of  knowledge  within  which  simulation  models  can  be  constructed, 

"we  can  create  an  artifically  intelligent  'expert'  system  which  can  patiently 
provide  the  student  with  a logically  deep  sounding  board  for  his  own  ideas 
(Brown,  Burton,  & Bell,  1974)."  Subject  areas  within  which  simulation 
models  can  be  constructed  include  (Brown,  Burton,  & Bell,  1974): 


(1) 
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programming,  wherein  one  can  simulate  the  execution  of  a 
program  to  understand  how  it  works, 

(2)  areas  of  medical  science  within  which  physiological  or 
pharmacological  processes  can  be  modeled, 
and  (3)  areas  of  engineering  within  which  physical  systems  can  be  modeled. 

Work  by  Collins  and  associates  suggests  that  powerful  inferential 
mechanisms  may  also  be  available  in  subject  areas  whose  content  is  not 
adaptable  to  partial  or  total  representation  by  simulation  models.  These 
investigators  have  developed  procedures  which  search  a semantic  network 
a la  Carbonell  in  order  to  derive  Implicit  facts  consistent  with  or  suggested 
by  the  explicit  information  which  the  network  houses.  Collins  and  associates 
are  working  with  SCHOLAR-like  tutors  and  have,  in  fact,  continued  work  on 
Carbonell' s geography  tutor,  now  called  GEO-SCHOLAR.  They  have  also  developed 
SCHOLAR-like  tutors  of  the  ARPA  network  (Grignetti  & Warnock,  1973)  and  a 
text-editing  system  called  NLS  (Grignetti  et  al . , 1974).  Before  discussing 
the  inferential  procedures  that  Collins  and  coworkers  have  modeled,  a descrip- 
tion of  the  purpose  and  appearance  of  SCHOLAR'S  tutorial  interaction  will 
be  given.  GEO-SCHOLAR  is  chosen  to  exemplify  the  family  of  SCHOLAR-like 
tutors  and  will  be  the  focus  of  our  discussion. 

GEO-SCHOLAR  can  handle  unanticipated  student  dialogue  in  a subset  of 
English  which  is  constrained,  for  the  most  part,  by  SCHOLAR'S  limited  syntactic 
capabilities.  The  content  of  the  dialogue  is  dependent  on  student  responses, 
questions,  and  requests.  SCHOLAR  uses  its  semantic  network  to  generate  the 
text  it  presents,  the  questions  it  asks,  and  the  responses  it  constructs 
to  both  its  own  questions  and  those  posed  by  the  student.  SCHOLAR  can  call 
upon  programmed  procedures  to  make  several  types  of  calculations  and 
inferences. 

Figure  2 presents  a sample  protocol  of  an  interaction  with  GEO-SCHOLAR. 

It  is  taken  from  Collins  and  Warnock  (1974). 
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Figure  2 


A GEO-SCHOLAR-student  interaction.  The  protocol  was  originally  presented 
in  Collins  and  Warnock  (1974).  Statements  entered  by  the  student  are 
underlined . 


WHAT  IS  THE  LANGUAGE  OF  ARGENTINA? 

PROBABLY  SPANISH. 

VERY  GOOD. 

APPROXIMATELY  WHAT  IS  THE  AREA  OF  ARGENTINA? 

TELL  ME  SOMETHING  ABOUT  PERU. 

PERU  IS  A COUNTRY.  IT  IS  LOCATED  IN  SOUTH  AMERICA.  THE  CAPITAL  OF  PERU  IS  LIMA. 

THE  AREA  OF  ARGENTINA  IS  1,100,000  SQUARE  MILES . 

APPROXIMATELY  TRUE,  BUT  A MORE  EXACT  VALUE  IS  1,200,000. 

THE  EASTERN  BORDERING  COUNTRY  OF  PERU  IS . 

CATHOLICISM. 

WRONG.  CATHOLICISM  IS  NOT  A COUNTRY  BUT  A RELIGION.  THE  CORRECT  ANSWER  IS: 
BRAZIL.  WHERE  IS  PERU? 

WHAT  IS  THE  MOST  IMPORTANT  DIFFERENCE  BETWEEN  PERU  ANE  PARAGUAY? 

THE  PRIMARY  DIFFERENCE  IS:  THE  POPULATION  OF  PERU  IS  LARGER  BY  A FACTOR  OF  7.8. 


Collins  and  associates  have  attempted  to  extend  SCHOLAR'S  information 
base  beyond  that  which  is  literally  stored  in  the  semantic  network  in  order 
to  broaden  both  the  range  of  knowledge  about  geography  which  SCHOLAR  can 
talk  about  and  respond  to,  and  the  types  of  knowledge  SCHOLAR  can  effectively 
teach.  They  began  this  effort  by  classifying  the  inferential  strategies 
used  by  people  (Carbonell  & Collins,  1°73).  Several  of  the  strategies 
identified  have  been  implemented  as  procedures  used  by  SCHOLAR  (Collins  N 
Warnock,  1974;  Collins,  Warnock,  Aiello  4 Miller,  1975). 


To  illustrate  the  kinds  of  inferences  SCHOLAR  can  make,  several  examples 
will  be  described.  The  examples  are  presented  in  a positive,  fail  safe 
manner  but  the  reader  should  keep  in  mind  that  the  success  of  an  inference 
depends  on  the  presence  and  appropriateness  of  facts  in  the  network  upon 
which  the  inference  rests. 


(1)  When  a property  is  not  on  the  property  list  of  a concept, 
SCHOLAR  can  still  determine  whether  the  concept  has  the 
property.  For  example,  to  answer  the  question  "Does  the 
Llanos  have  a rainy  season?"  SCHOLAR  first  searches 
through  the  Information  directly  linked  with  the  Llanos. 

With  the  present  data  base,  it  finds  nothing  about  a 
rainv  season.  It  does  find,  though,  that  the  Llanos  is 
a savanah . Searching  through  the  property  list  of 
"savanah",  SCHOLAR  finds  that  a savanah  has  a rainy 
season.  SCHOLAR  then  infers  that  the  Llanos  must  also. 

(2)  SCHOLAR  can  determine  whether  a question  is  true  or 
false.  For  example,  to  answer  the  question  "Is  Bolivia 
a capital?"  SCHOLAR  observes  that  Bolivia  is  a country 
and  that  a capital  is  a city.  It  also  has  stored  the 
fact  that  a city  and  a country  are  different  kinds  of 
places.  Hence  SCHOLAR  answers  "no". 

(3)  SCHOLAR  can  determine  the  agricultural  products  or  climate 
of  a geographical  region.  For  example,  to  answer  the 
question  "Is  the  climate  of  Buenos  Aires  subtropical?"  SCHOLAR 
first  checks  the  property  list  of  Buenos  Aires  but  finds 
no  mention  of  it  being  subtropical  . SCHOLAR  then  attempts 
to  locate  a city  which  is  subtropical.  It  finds  Caracas. 
SCHOLAR  then  checks  whether  the  latitude  and  longitude  of 
Caracas  are  similar  to  those  of  Buenos  Aires.  They  are 
not.  SCHOLAR  infers  that  it  is  unlikely  that  Buenos  Aires 
is  subtropical. 

The  strategy  SCHOLAR  emplovs  to  answer  questions 
about  products  and  climate  depends  on  SCHOLAR'S  endowment 
of  knowledge  about  the  determinants  of  products  (climate, 
rainfall,  and  soil  fertility)  and  climate  (longitude  and 
latitude).  It  also  depends  on  the  general,  and  sometimes 
misleading,  rule  that  if  cer  ain  criteria  of  similarity 
are  met  between  two  situations  then  a result  that  pertains 
to  the  first  pertains  to  the  second. 


Using  inferences  like  those  illustrated  above,  SCHOLAR  can  now  talk 
about  information  that  is  not  explicitly  stored  in  the  network.  This 
power  has  greatly  enhanced  the  system's  ability  to  interact  with  students 
both  by  enlargening  the  range  of  student  inputs  It  can  intelligently 
address  and  by  increasing  the  depth  of  its  responses. 

The  networks  used  by  SCHOLAR  and  SOPHIE  are  instances  of  a class  of 
structures  known  as  semantic  networks.  As  the  name  implies,  a semantic 
network  attempts  to  represent  the  meaning  of  facts.  Typically  links 
associate  the  facts  into  a totally  connected  structure.  In  the  case  of 
SCHOLAR,  for  example,  attributes  can  be  thought  of  as  the  links  of  the 
network.  Other  examples  of  semantic  networks  can  be  found  in  the  works 
of  Anderson  and  Bower  (1973),  Norman  and  Rumelhart  (1975),  and  Simmons 
(1973),  while  a discussion  of  fundamental  issues  is  presented  bv  Woods 
(1975) . 

The  work  of  Norman  and  Rumelhart  (1975)  (and  also  the  work  of  Anderson 
and  Bower,  1973)  on  semantic  networks  is  the  product  of  an  attempt  to  model 
human  memory.  Because  of  this,  a discussion  of  the  Norman-Rumelhar t model 
will  be  given  in  the  next  section  on  modeling  the  learner.  However  we 
would  like  to  briefly  discuss  at  this  point  an  extension  of  their  modeling 
effort  into  the  realm  of  representing  complex  subject  matter. 

Semantic  networks  as  typically  designed  readily  represent  factual  know- 
ledge, but  confront  a more  difficult  problem  in  attempting  to  model  the 
higher  level  conceptualizations  which  exist  in  most  subject  areas.  For  example, 
"it  is  easy  to  represent  that  General  Grant  smoked  cigars  or  that  the  Amazon 
River  had  numerous  tributaries,  but  not  easy  to  represent  that  the  North's 
plan  of  action  in  the  Civil  War  was  to  cut  off  the  South  from  external 
supplies,  and  that  this  plan  motivated  Lincoln  to  keep  a major  segment  of 
troops  at  Cairo,  Illinois  and  St.  Louis,  and  that  it  eventually  led  Grant 
to  cut  off  the  Tennessee  and  Cumberland  rivers  (Norman,  ]97b)." 

Drawing  from  work  by  Rumelhart  (1°75,  1^76)  on  representing  simple 
stories,  Norman  and  Rumelhart  (Norman,  1^76)  are  developing  a scheme  to 
represent  complex  episodes,  like  the  Civil  War  episode  given  above.  A 
tutor  of  history  is  presently  under  development  bv  these  investigators 
that  will  provide  a vehicle  within  which  their  representation  ideas  can 
be  Implemented  and  evaluated.  Although  the  resulting  model  will  apply 
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directly  to  historical  narratives,  its  development  may  also  provide  some 
understanding  of  how  to  structure  the  complex  interrelations  which  comprise 
other  subject  areas. 

Modeling  the  Learner's  State  of  Knowledge 

As  was  suggested  in  the  previous  section,  if  a CA1  program  has  access 
to  information  tnat  endows  it  with  some  subject  matter  expertise,  decisions 
about  what  material  to  present  can  he  made  dynamically  as  the  student 
progresses  through  the  course.  Tutor-tutee  dialogue  thus  has  the  potential 
to  he  keyed  to  the  needs  of  the  student.  Rut  a model  of  its  subject  domain 
provides  only  part  of  the  data  the  tutor  may  require  to  respond  adaptively 
to  the  student;  the  tutor  may  also  require  information  about  the  student. 

As  Norman  (1973)  has  noted,  without  a model  of  the  knowledge  of  its 
student,  a computer  based  system  that  supports  interactive  dialogue  has 
only  reached  the  stage  of  egocentric  intellectual  development. 

Three  kinds  of  models  of  the  learner  which  mav  be  implemented  within 
a tutor  will  he  illustrated.  Thcv  differ  primarily  in  how  they  function 
within  the  tutor.  We  discuss  first  models  employed  hy  the  tutor  to 
diagnose  student  errors.  Such  models  typically  depend  on  taxonomies  of  errors 
the  student  mav  make.  We  discuss  next  models  employed  to  record  what  the 
student  has  learned  about  the  subject  area.  Such  models  typically  depend  on 
explicit  representations  of  those  parts  of  the  subject  matter  the  student  has 
mastered.  And  finally,  we  discuss  models  employed  to  determine  the  manner  in 
which  information  should  be  presented.  Such  models  are  typically  operational 
statements  of  how  students  acquire  knowledge. 

Models  Used  For  Diagnostics 

Assuming  a student's  answer  to  a question  requires  intermediate 
steps  for  its  derivation,  when  the  answer  is  wrong  the  tutor  can  choose 
to  attend  to  the  answer,  its  derivation,  or  both.  For  example,  if  the 
tutor  acknowledges  that  the  answer  is  incorrect  and  provides  the  right 
answer,  the  tutor  has  attended  to  the  product — the  answer.  On  the  other 


-4 


- 1 

hand,  attention  to  the  derivation  is  achieved  if  the  tutor  indicates  faults 
in  the  student's  reasoning  or  begins  to  question  the  student  about  the 
manner  in  which  he  arrived  at  his  answer. 

The  tutor  can  attend  to  the  derivational  process  without  concern  for 
the  particular  process  employed  bv  the  student.  For  example,  Wexler's 
(1970)  geography  tutor  responded  to  a student's  incorrect  responses  by  out- 
putting  the  derivational  path  the  computer  would  have  considered  had 
it  been  asked  the  question.  But  this  form  of  remediation  is  not  tailored 

I ; 

to  the  weaknesses  of  the  student. 

If  remediation  is  to  be  keyed  to  the  student's  derivational  process, 
the  tutor  must  turn  to  procedures  which  recognize  faults  in  the  student's 
reasoning.  The  process  of  recognizing  faults  is  complicated  by  the  fact 
that  it  is  often  unnatural  for  a student  to  detail  every  step  he  has 
taken  to  arrive  at  his  answer.  Hence  the  procedures  may  have  to  infer 
| mistakes  in  reasoning  from  a,  perhaps,  sketchy  account  of  the  steps  the 

student  has  taken.  Whatever  inference  strategies  are  employed  though,  the 
process  of  diagnosing  student  faults  is  likely  to  depend  on  the  presence 

■ 

within  the  svstem  of  a taxomomy  of  potential  errors  the  student  may  make 

(Carbonell,  1970b;  Siklossv,  1970).  ^ 

Brown  and  coworkers  employ  taxonomies  in  both  a tutor  which  provides 
guidance  while  a student  plays  a game  (Burton  & Brown,  1976)  and  a tutor 
which  provides  guidance  while  a student  solves  algebraic  equations 
(Brown,  1975a,  b) . The  game  tutor  points  out  weaknesses  in  the  strategy 

I* 

a student  employs  to  plav  the  PLATO  game  "How  the  West  Was  Won",  a 
game  supporting  drill  and  practice  In  elementary  arithmetic.  The  algebra 
tutor  helps  a student  acquire  algebraic  skills  by  presenting  him  with 
problems  and  providing  him  with  feedback  relating  to  his  solutions  of  the 
prohl ems . 

* 

This  game  was  written  bv  Bonnie  Anderson  for  the  PLATO  Elementary 
Mathematics  Project. 
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The  algebra  tutor  bases  its  diagnostics  of  problems  incorrectly  solved 
on  a taxonomy  of  errors  which  reflect  universal  kinds  of  misunderstandings 
related  to  algebraic  manipulations.  A subset  of  the  taxonomy  the  tutor  uses 
is  given  in  Brown  (1975a) . The  taxonomy  used  by  the  game  tutor  is  not  really 
a taxonomy  of  errors,  but  a taxonomy  of  strategies  or  basic  operations  the 
student  may  employ  to  play  the  game.  The.  diagnostic  module  calls  on  this 
taxonomy  of  strategies  to  note  wiat  operations  the  student  employs  in  his 
moves  and  what  operations  the  student  could  emplov  to  improve  his  game. 

In  both  tutors,  remediation  is  based  on  weaknesses  which  the  diagnostic  pro- 
cedures uncover. 

Both  tutors  reiy  on  records  of  past  performance  to  more  accurately  de- 
termine the  weaknesses  in  a student's  performance.  Student  histories  are 
used  to  cope  with  the  difficulty  suggested  earlier,  namely  the  difficulty 
in  inferring  faults  based  on  only  a few  isolated  student  inputs.  In  the 
game  tutor,  a student's  single  move  typically  does  not  provide  enough 
information  to  conclude  that  he  is  unaware  of  some  basic  strategy  that 
would  improve  his  game.  The  game  tutor  calls  upon  a summary  of  the  student's 
previous  moves  to  aid  in  the  inference  of  deficiencies.  Similarly,  in  its 
attempt  to  ascertain  the  improper  operations  the  student  has  used  to  solve 
a problem,  the  algebra  tutor  turns  to  a summary  of  problem  solving  techniques 
presently  in  the  student's  repertoire.  Included  in  the  summary  are  both  the 
legitimate,  heur istically  sound  transformations  known  to  the  student  and  the 
illegal  or  counter  productive  transformations  he  has  employed  and  may 
employ  again. 


Models  Used  To  Record  Student  Knowledge 

The  summaries  used  bv  Brown's  tutors  to  aid  in  diagnosing  student 
weaknesses  are  instances  of  models  which  record  the  state  of  a student's 
knowledge.  In  the  case  of  the  game  tutor  the  student's  state  of  knowledge 
is  defined  bv  ills  repertoire  of  strategies;  in  the  case  of  the  remedial 
algebra  tutor,  the  student's  state  of  knowledge  is  defined  by  his  repertoire 
of  algebraic  manipulations.  Models  which  reflect  a student's  present  under- 
standing oi  the  subject  area  can  provide  data  influencing  a number  of 
tutor  activities,  such  as  decisions  concerning  what  to  teach  next  or  the  depth 
of  remediation.  Such  models  can  be  used  to  structure  the  tutor's  explana- 
tions in  terms  familiar  to  the  student.  They  can  also  provide  data  pertinent 
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to  determining  why  a student  has  asked  a specific  question,  or,  as  we  have 
seen  in  the  section  on  diagnostics,  determining  why  a student  has  incorrectly 
answ  red  a question. 

As  in  research  on  modeling  the  subject  matter,  a variety  of  structures 
which  model  the  student's  present  state  of  knowledge  have  been  proposed. 
Indeed,  the  two  modeling  efforts  are  Intimately  connected.  Methods  used 
to  model  the  subject  matter  may  be  applicable  to  modeling  the  student's 
state  of  knowledge.  For  example.  Self  (1174)  has  proposed  that  some  forms 
of  knowledge  can  be  readily  represented  as  programs.  While  the  representation 
of  the  subject  domain  can  consist  of  programs  which,  when  computed,  result 
In  the  "ideal"  answers,  those  which  model  the  student  produce  answers  which 
match  those  the  student  would  supply. 

The  model  of  the  subject  area  itself  may  provide  a natural  depository 
for  data  regarding  what  the  student  has  learned.  Both  Wexler  (1970)  and 
Carbonell  (1970a,  b)  suggested  flagging  their  networks  in  order  to  keep  track 
of  areas  the  student  had  mastered.  Atkinson  and  associates  have  also  taken 
this  approach  in  their  tutor  of  BASIC  known  as  BIP  (Barr,  Beard,  & Atkinson, 
1975a,  1975b:  Beard,  Barr,  Fletcher,  & Atkinson,  1975). 

BIP  (BASIC  instructional  Program)  is  an  interactive  problem-solving 
laboratory  that  offers  tutorial  assistance  to  students  solving  introductory 
programming  problems.  The  system  is  a stand-alone,  fully  self-contained 
course  in  BASIC  programming.  The  emphasis  of  the  course  is  on  solving 
programming  problems.  To  this  end  students  are  presented  a series  of 
programming  problems  which  they  solve  on-line.  As  the  student  develops  a 
program  he  is  directed  to  appropriate  sections  of  a hard-copy  manual  which 
explains  such  things  ns  BASIC  statements  and  programming  structures.  The 
student  is  al«o  encouraged  to  use  numerous  on-line  student-oriented  features 
such  as  interactive  debugging  facilities  and  help  options. 

To  aid  in  decisions  concerning  the  sequencing  of  programming  problems, 

BIP  calls  upon  a Curriculum  Information  Network  (CTN)  . A CIN  is  a 
structured  representation  of  the  curriculum  which  consists,  in  part,  of  the 
problems  or  tasks  the  tutor  mav  present  to  the  student,  linked  to  the  basic 
skills  required  for  successful  completion  of  the  problems.  BIP  uses  this 
network  to  flag  the  skills  its  student  has  mastered.  Presently  each  skill 
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is  tagged  with  an  array  of  counters  which  note  the  student's  state  of 
acquisition  of  the  skill.  The  counter  values  are  determined  by  the 
student's  ability  to  complete  programming  problems  which  require  the 
skill  and  the  student's  own  assessment  of  his  mastery. 


Models  of  the  Process  of  Acquisition 

Models  of  how  students  acquire  knowledge  are  basic  to  the  understanding  of 
how  information  should  be  presented.  If  operational  models  were  available, 
the  tutor  would  have  a powerful  means  for  determining  the  structure  of  its 
tutorial  interaction.  In  the  area  of  paired-associate  learning,  a variety 
of  mathematical  models  exist  which  describe  probablistical ly  how  information 
is  acquired  (Fletcher,  1975),  and,  thus,  suggest  how  content  should  be 
presented.  But  for  more  complex  learning  adequate  operational  statements 
are  lacking.  The  majority  of  published  work  related  to  the  learning  process, 
while  intellectually  rewarding,  is  of  little  assistance  in  defining  precise 
models.  But,  progress  on  this  front  is  being  made  by  Norman  and  associates 
at  the  Center  for  Human  Information  Processing  at  the  University  of  California. 

Norman  and  associates  are  engaged  in  modeling  human  memory.  Their  work 
has  recently  been  described  in  the  collection  of  papers:  Explorations  into 
Cognition  (Norman  & Kumelhart,  J975).  At  the  foundation  of  the  memory  model 
employed  by  Norman  et  al.  is  a set  of  general  functions  which  specify  the 
relations  which  exist  among  subsets  of  concepts.  The  scheme  for  functional 
representation  of  related  concepts  borrows  heavily  from  Fillmore's  (1968) 
notion  of  "case  grammar"  in  that  it  assigns  particular  roles  to  the  concepts 
of  a proposition. 

The  investigators  are  attempting  to  determine  a primitive  set  of 
general  functions  so  that  all  propositions  can  be  expressed  in  terms  of 
some  subset  of  these  more  basic  relations.  So  far  their  work  has  concentrated 
on  describing  functions  that  correspond  to  the  meanings  of  verbs  and  the 
primitives  which  underlie  verbal  structures. 

To  illustrate  the  type  of  model  Norman  and  coworkers  are  exploring 
we  will  consider  the  verb  "give".  The  meaning  of  "give"  that  corresponds 
to  one  person  placing  an  object  in  the  possession  of  another  person  is 
represented  bv  the  following  function: 

give  [agent,  object,  recipient,  time) 


18  - 


The  arguments  of  the  function  suggest  the  set  of  restrictions  that  apply 
to  the  values  that  can  be  used  to  fill  the  arguments.  Thus  the  first 
argument — agent — denotes  an  animate  being  capable  of  instigating  the  action. 
This  meaning  of  "give"  can  be  further  analyzed  by  considering  the  primitive 
notions  which  underlie  the  action,  such  as  the  notion  of  one  event 
causing  another  and  the  notion  of  an  object  being  transferred  from  some 
source  to  some  goal. 

The  memory  system  Norman  and  coworkers  have  proposed  adapts  readily  to 
a variety  ol'  cognitive  tasks,  including  information  acquisition  (Rumelhart  & 
Norman,  1975),  language  comprehension  (Rumelhart  & Levin,  1975)  and 
problem  solving  (Eisenstadt  & Karcev , 1975).  it  is  not  yet  known  whether 
the  structures  and  processes  modeled  parallel  human  structures  and  processes, 
but  the  success  of  the  model  should  encourage  research  in  this  area.  Norman 
and  associates  have  already  pursued  some  research  along  this  line  (e.g., 

Gent  nor,  1975). 


Modeling  Teaching  Strategies 

CAI  tutors  in  which  the  coding  of  procedures  which  control  the 
flow  of  the  instructional  interaction  Is  distinct  from  other  modules  of 
the  tutor,  such  as  the  student  or  subject  matter  models,  provide  convenient 
tools  for  exploring  various  teaching  scenarios.  Tvpicallv  the  outward  behavior 
of  these  tutors  can  be  completely  redesigned  in  just  a matter  of  days.  Hence 
the  instructional  researcher  has  at  his  disposal  a well-controlled  laboratory 
within  which  to  model  instructional  strategies,  implement  the  models,  and 
compare  their  results  in  trrms  of  learning  achievement. 

Separate  control  procedures  exist  in  varying  degrees  in  the  systems 
being  explored  at  BBN  by  Brown  and  Collins  and  in  Atkinson's  tutor  of  BASIC, 
BIP.  Investigations  concerned  with  isolating  optimal  instructional  strategies 
are  being  pursued  at  both  sites. 

Collins  and  associates  have  engaged  in  modeling  teaching  strategies  for 
their  tutor  of  geography,  GEO-SCHOLAR  (Collins,  1976;  Collins,  Warnock  N 
Passat l ume , 1975).  To  isolate  teaching  strategies  employed  by  human  tutors, 
Collins  and  cottorke rs  analyzed  human  tutor-tutee  teaching  sessions.  Based 
on  these  analyses  Collins  and  associates  have  proposed  several  hypotheses 
about  how  the  tutor  relates  his  teaching  to  the  individual  student.  The 
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investigators  have  implemented  some  of  these  hypotheses  as  strategies  which 
GEO-SCHOLAR  can  incorporate  into  its  tutorial  interactions.  Experiments  have 
been  conducted  to  define  those  characteristics  of  implemented  strategies  which 
facilitate  learning  relative  to  a frame-oriented  tutorial  (Collins,  1974), 

Research  suggested  by  Brown  and  coworkers  (e.g..  Brown,  1975b;  Brown 
N Burton,  1975)  using  tutors  of  problem  solving  like  SOPHIE  and  the  remedial 
algebra  tutor  promises  to  provide  data  concerning  optimal  strategies  for 
teaching  problem-solving  techniques.  Because  of  the  computer's  ability 
to  search  a solution  space  much  more  efficiently  and  effectively  than  can 
a human  tutor,  some  strategies  which  can  be  studied  easily  in  these  problem- 
solving laboratories  would  be  difficult,  if  not  impossible,  to  model  and 
evaluate  in  a noncomputerized  setting. 

Atkinson  and  his  coworkers  at  the  Institute  for  Mathematical  Studies 
in  the  Social  Sciences  are  exploring  the  use  of  curriculum  networks  to 
provide  data  for  strategies  which  sequence  instruction.  By  investigating 
problem  sequencing  as  a function  of  student  progress  in  skill  acquisition, 
an  attempt  is  being  made  to  model  optimal  instructional  sequencing  strategies 
over  the  network  (Barr,  Beard,  & Atkinson,  1975b). 

CONCLUSIONS 

The  military  must  train  a large,  heterogeneous  student  population. 

Because  the  population  is  so  varied,  adaptive  instructional  environments 
are  required  for  manv  training  objectives.  Because  of  the  large  number  of 
students  involved,  such  environments  cannot  be  provided  using  labor  intensive 
traditional  approaches.  By  providing  knowledge  concerning  the  development 
of  responsive  automated  tutors,  AKFA  supported  projects  have  addressed  these 
issues . 

One  large  training  area  where  cost-effective  applications  of  responsive 
tutors  seems  most  promising  is  tutorial  simulations  of  laboratory  situations. 

The  traditional  laboratorv  has  severe  limitations.  Instructors  may  not  have 
the  time  to  adequately  pursue  a student's  hypotheses  or  questions,  or 
evaluate  his  procedures;  in  laboratories  requiring  measurements,  a student's 
rime  wav  b»  ineffectually  lost  in  preparations  which  add  little  to  the 
training  experience.  If  expensive  equipment  is  being  used  or  expended,  as 
in  combat  training,  the  costs  of  training  mav  severely  constrain  its  effec- 
tiveness. Responsive  CAI  promises  to  eliminate  these  threats  to  learning 
effectiveness  while  possibly  decreasing  training  costs  by  providing  an  environment 
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in  which  the  student  may  manipulate,  with  "expert"  supervision,  a 
computer  model  of  the  system  under  investigation. 

As  test  beds  for  instructional  research,  the  systems  developed  by  ARPA 
supported  programs  will  aid  in  isolating  software  capabilities  which 
facilitate  training  objectives.  Work  must  then  turn  toward  generalizing 
these  capabilities  to  insure  their  transportability  to  other  CAI  environments. 
Also,  as  vehicles  for  instructional  research,  the  systems  will  aid  in 
identifying  and  rigorously  defining  optimal  instructional  strategies,  thus 
providing  guidelines  for  authors  of  CAT  materials. 

TRENDS 

Advances  in  natural  language  comprehension  will  facilitate  both  the 
student-system  and  author-system  interfaces,  and,  in  turn  increase  the 
range  of  training  objectives  which  can  be  achieved  in  a CAI  environment. 

As  a CAI  system  becomes  less  responsive  to  natural  language  input, 
the  number  of  extraneous  skills  a student  must  develop  to  interact  with  the 
program  increases.  The  syntax  of  acceptable  input  statements  becomes  more 
prescribed,  the  semantics  more  constrained.  In  the  framework  of  CAI,  research 
in  natural  language  processing  will  reduce  interaction  difficulties  by 
decreasing  the  extent  to  which  a student  must  process  the  system's 
language  while  increasing  the  extent  to  which  the  system  processes  the 
student's.  Also  because  a broader  range  of  student  responses  will  be 
"understood"  by  the  computer,  the  extent  to  which  the  system  author  must 
anticipate  and  constrain  student  inputs  will  decrease,  encouraging  the 
development  of  more  sophisticated  training  environments. 

Rv  providing  the  system  author  with  the  opportunity  to  interact  with 
the  computer  in  natural  language,  the  time  needed  to  create  CAI  materials 
and  the  system  limits  imposed  on  those  materials  by  programming  language 
constraints  will  decrease.  Presently  much  time  is  invested  in  learning 
programming  skills  and  format  convent  ions . The  programming  of  complex 
CAI  materials  requires  considerable  programming  experience  and,  hence,  the 
availability  of  expert  programmers.  Rv  providing  the  CAI  author  with  a 
more  natural  interface,  time  needed  to  learn  computer  conventions  and 
dependence  on  expert  programming  skills  will  decrease. 
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The  capabilities  being  developed  in  responsive  CAI  tutors  will  be  of 
value  as  job-performance  aids.  For  example,  BIP's  student-program 
interpreter  contains  debugging  aids  that  would  be  of  assistance  to 
programmers  outside  of  the  classroom  environment  (Barr  & Beard,  1976). 
Also  the  simulation  models  developed  for  responsive  tutors  may  be  useful 
to  authors  of  more  traditional  dr ill-and-prac tine  or  tutorial  programs. 
Such  models  can  quickly  provide  the  author  with  data  which  would  be 
time-consuming  to  collect  by  other  means. 
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